Citation Data Is Subtle Stuff. A Primer on Evaluating a Scientist's Performance

When starting to compile citation data from the scientific literature over 25 years ago, I aimed to create a new tool for information retrieval—the Science Citation Index (SC~. Out of this came a useful by-product: a huge and ever-increasing database containing indicators of intellectual connections among scientists and their publications. The SCZattracted the attention of historians and sociologists of science and served as a catalyst to the field of scientometrics, which uses quantitative methods to analyze the process and development of science. The 1S1 database has facilitated large-scale quantitative studies of the scientific performance of countries, institutions, fields, departments and individuals. In recent years, such scientometric studies have even contributed to public policy decisions in science. However, it is in assessing the performance of individual scientists, especially in the context of promotion or grant decisions, that the use of citation data is most controversial. This controversy has arisen for a number of reasons—none more basic perhaps than a common aversion to the impersonal judgment of numbers. Beyond that, however much of the criticism of citation measures is a response to “quickand-dirty” citation-counting, a thoughtless practice that reveals little more than an amateur at work and gives citation analysis a bad name. I have repeatedly warned about the mistaken conclusions drawn from the crude manipulation of citation data. Recently (The Scientist, February 23, 1987, p. 9), I emphasized the distinct difference between “the simple-minded counting of articles or citations as indicators of quality and the in-depth analysis that can and should be carried out.” This statement warrants some expansion. Here, then, is a primer on how to evaluate scientists using citation data. First, you’ll need a complete and accurate bibliography of the candidate’s publications. The SCl Citation Index lists all cited papers and books under the first author’s name. Using the bibliography or full CV will ensure that all coauthored publications are included in your collection of citation data. (If a bibliography or CV is unavailable, a search through successive years of the SCYSource Index, which cross-references secondary to primary authors, can provide a reasonably complete list of papers.)